Overview

This PR introduces significant modifications to the GitHub Action for testing Open edX plugins in the Tutor environment. These changes are necessary for improving the effectiveness and reliability of our integration tests

Motivation for Changes

The initial implementation ran integration tests inside the LMS container of Tutor. This approach created several challenges:

• Networking Issues: Running tests inside the same container restricted the ability to make requests to external hosts, leading to unreliable test outcomes.
• Realistic Testing: To ensure that our integration tests reflect real-world interactions between our plugins and the Open edX platform, it is crucial to run them outside the Tutor container, maintaining a separation between the tests and the platform environment.

To address these issues, this PR modifies the workflow to run integration tests outside the Tutor container, improving the testing environment and results.

Workflow Steps and Their Necessity

1. Set Tutor Environment Variables:

   • This step sets necessary environment variables like LMS_HOST, CMS_HOST, and paths for Tutor and plugins. These variables are essential for proper configuration and operation of Tutor.

2. Install and Prepare Tutor:

   • Installs the specified version of Tutor, configures it with the necessary host settings, and launches Tutor in non-interactive mode. This ensures that the Tutor environment is ready for testing.

3. Configure Caddyfile and Open edX Settings:

   • Creates a patches.yml file to customize the Caddyfile and Open edX production settings. This configuration allows for the necessary routing for testing with multiple sites and security settings for our tests.

4. Install Plugin and Extra Requirements:

   • Installs the specified plugin and any additional dependencies required for testing. This ensures that the environment is properly set up for the plugin being tested.

5. Run Migrations:

   • Executes database migrations for both the LMS and CMS. This step is crucial to ensure that the database schema is up-to-date with the changes introduced by the plugins and other dependencies.

6. Load Initial Data for the Tests:

   • Optionally copies fixtures into the LMS container and loads them into the database. This allows tests to run with pre-defined data, ensuring consistency and reliability.

7. Curl Heartbeat:

   • Verifies that the LMS is running and responsive by checking the heartbeat endpoint. This step helps ensure that the environment is fully operational before running tests.

8. Run Integration Tests:

   • Executes the specified shell script to run the integration tests. This step is where the actual testing takes place, utilizing the environment configured in previous steps.

Personalization Options

This workflow includes several inputs that can be customized:

• app_name: The name of the application to test (e.g., eox-core).
• tutor_version: The version of Tutor to use, allowing for testing against different Tutor releases.
• shell_file_to_run: The path to the shell script that contains the integration tests.
• openedx_extra_pip_requeriments: Optional additional dependencies needed for testing.
• fixtures_file: Optional path to a JSON file containing initial data to load for the tests.

Testing the Workflow

Steps to test this workflow:

1. You can create a branch from any of the EOX plugins. For example, create a branch from bav/add-integration-tests in eox-core. I’m using this branch as an example because it includes several integration tests.

2. Add the script that installs the requirements and runs the tests, for example:

   #!/bin/bash
   
   bgreen () { printf "\n\n\e[1m\e[32m" ; $@ ; printf "\e[0m\n\n"; }
   
   bgreen echo "Installing test requirements"
   make install-dev-dependencies
   
   bgreen echo "Running integration tests"
   pytest eox_core/api/v1/tests/integration/test_views.py

3. Ensure that the workflow calls the action with the corresponding inputs and uses the branch mjh/run-integration-tests-outside-container. Here’s an example of the file that executes the workflow:

   name: Tutor Integration Tests
   on: [pull_request]
   
   jobs:
     integration-test:
       name: Tutor Integration Tests
       runs-on: ubuntu-latest
       strategy:
         matrix:
           tutor_version: ['<18.0.0', '<19.0.0']
       steps:
         - name: Run Integration Tests
           uses: eduNEXT/integration-test-in-tutor@mjh/run-integration-tests-outside-container
           with:
             tutor_version: ${{ matrix.tutor_version }}
             app_name: 'eox-core'
             openedx_extra_pip_requirements: 'eox-tenant'
             shell_file_to_run: 'scripts/integration.sh'
             fixtures_file: 'fixtures/initial_data.json'

4. Check that on-push, the integration tests run successfully, as shown in this test branch: test: add integration tests eox-core#278

⭐ Important:

This PR needs to be updated with the implementation from PR #3 to support running the tests in Tutor nightly. The update will be made once PR #3 is merged.